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Abstract — Risk identification during the early design phases of 
complex systems is commonly implemented but often fails to 
result in the identification of events and circumstances that truly 
challenge project performance. Inefficiencies in cost and schedule 
estimation are usually held accountable for cost and schedule 
overruns, but the true root cause is often the realization of 
programmatic risks. A deeper understanding of frequent risk 
identification trends and biases pervasive during space system 
design and development is needed, for it would lead to improved 
execution of existing identification processes and methods. 
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IV. Introduction 

A commonly stated problem across space system design and 
development efforts is that severe deficiencies exist within 
system cost and schedule estimating methodologies. Although 
such criticism is valid to an extent, the assignment of causality 
from flawed cost and schedule estimating efforts to subsequent 
cost and schedule overruns often overlooks deeper seated root 
causes. Oftentimes, the system design itself, from a mass, 
power, and performance perspective, has significantly evolved 
from the initial design used for cost and schedule estimating 
purposes. One can argue that such growth should have been 
accounted for by the inclusion of adequate margin within 
performance, cost, and schedule planning. However, such 
planning requires some amount of anticipation of the types of 
challenges that a development project will face during the 
project life cycle. Risk management, as a sub-discipline to 
project systems engineering efforts, is typically tasked with 
anticipating and managing such challenges. If the world was 
one in which the same set of challenges and threats recur 
within each development effort, the likelihood and impact of 
these events could easily be assessed and protected against. 
Instead, each project is prone to its own set of challenges - 
some foreseeable, some not - that risk managers struggle to 
anticipate. When unanticipated events do happen of enough 
impact that they surpass budgeted reserves, they can quickly 
disrupt well formulated plans to the point of significant cost 
and schedule growth. For example, a recent NASA 
development project experienced a slip in launch date due to 
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budget reductions (foreseeable) that resulted in even longer 
delays when the launch service provider had difficulty in re- 
allocating their launch amidst crowded manifests (arguably 
unforeseeable) [1]. Because of this, this paper suggests that 
limitations within risk identification efforts, particularly 
subjective biases related to likelihood and consequence 
expectations, can and do lead to significant cost and schedule 
overruns. 

This paper discusses the early findings of a research effort 
aimed at better understanding the biases that may exist in space 
system risk identification, and how they appear to impact 
project performance. The intent is to evaluate temporal trends 
of risks identified throughout the life cycle of numerous 
projects to understand how early, or how late, the true 
challenges facing project performance are identified. The 
outcome is intended to be statistical confirmation that such 
biases do exist, along with general heuristics that can be used 
by future project managers to better avoid under appreciating 
the types of risks that do make a difference. Building upon 
existing literature that suggests such biases exist, this research 
further aims to confirm these behaviors using actual historical 
project data from a variety of NASA space system projects. 
These results pertaining to NASA projects, being large scale 
U.S. Government funded efforts, are estimated to be indicative 
of complex projects across numerous other industries and 
sectors. To that end, findings and trends are broadly applicable 
to a variety of applications. 


V . Risk identification within nasa 

Current NASA procedural requirements define risk 
management as “a set of activities aimed at achieving success 
by proactively risk-informing the selection of decision 
alternatives and then managing the implementation risks 
associated with the selected alternative” [2]. This definition 
correctly indicates that risk management is an activity that 
begins at the beginning of a project (e.g., when selecting 
between alternatives) and continues throughout the life of 
project. In fact, emphasis is equally placed on early risk- 
informed design decisions as well as continuous management 


of such risks. Instead of choosing a design alternative and 
worrying downstream about what risks may or may not pose 
problems, the various risk implications are viewed as decision 
criteria and managed thereafter. NASA’s procedural documents 
define this approach as a combination of Risk Informed 
Decision Making (RIDM) and Continuous Risk Management 
(CRM) [2]-[4]. RIDM pertains to the early phase decision 
making aspect while CRM is the implementation task of 
managing the identified risks, generally over the entire set of 
project life cycle phases [5]. One tool commonly used to 
implement the CRM philosophy is the risk matrix 
methodology. 

The NASA Systems Engineering Handbook defines the NASA 
project standard “5x5” risk matrix methodology, commonly 
used across the entire space sector, as a way to “combine 
qualitative and semi-quantitative measures of likelihood with 
similar measures of consequences” [6]. The strengths are that it 
is easily understood, can assist in tracking top-ranked risks at a 
high level, and can be easily used to communicate risk status 
information both within a project as well as to project 
stakeholders. However, there are numerous drawbacks - such 
as the fact that combinatorial interactions amongst multiple 
risks are not accounted for, aggregate risks are not quantified, 
uncertainty quantification is lacking, and general 
oversimplicity. At a high level, the standard risk matrix (Fig. 1) 
is partitioned into three sections based on the intersection of a 
risk’s estimated likelihood and consequence. Per the NASA 
Systems Engineering Handbook, a low (green) risk “has little 
or no potential for increase in cost, disruption of schedule, or 
degredation of performance” [6]. The moderate/medium 
(yellow) items “may cause some increase in cost, disruption of 
schedule, or degredation of performance”. Finally, the high 
(red) risk is “likely to cause significant increase in cost, 
disruption of schedule, or degredation of performance”. 
Although the matrix provides a useful way to convey how risks 
fall within the green, yellow, or red sections, as well as how 
they compare to one another in regards to likelihood and 
consequence, it falls short in providing context behind each 
risk. There is no clear way to capture the consequence of 
multiple risks being realized concurrently (i.e., the combined 
consequence could be greater than the sum of their individual 
consequences). Another drawback from the matrix approach is 
that one could infer that a risk with a likelihood of 2 and 
consequence of 4 is equal in impact (L x C) to one with a 
likelihood of 4 and consequence of 2, which may often not be 
the case. 

There are also further deficiencies of the standard 5x5 risk 
matrix documented in literature. For example, Cox points out 
that although their use is wide spread and commonly 
considered standard practice, there has been very little 
empirical evidence from studies showing their actual efficacy 
in managing risks [7]. That work points out several potential 
mathematical and logic flaws that result from the way risks are 
discretized and mapped; namely logic consistency, 
mathematical symmetry, and consistent coloring [7]. 
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FIGURE 1. NOTIONAL RISK MATRIX [6] 


Despite the documented limitations in using standard 5x5 risk 
matrices, this research leans heavily on the concept due to its 
preponderance amongst historical project documentation. For 
instance, each monthly NASA status document, regardless of 
which NASA Center is managing the project, routinely 
includes a 5x5 risk matrix containing the top 5-10 risks 
currently threatening the project at that time. Because such 
documentation is refreshed on a monthly basis it provides a 
rich data source for understanding temporal trends associated 
with these top risks. Combined with cost and schedule monthly 
status documentation, these risks trends provide insight into 
where biases potentially reside in project risk identification and 
tracking efforts. 

VI. Biases within risk identificaiton 
E. Risk Identificaiton Background 

Despite procedural requirements dictating that RIDM and 
CRM be used to actively identify and track various risks, the 
actual methodology used to identify the risks is not 
standardized. Each project typically has either a dedicated risk 
manager and/or risk management team, and it is up to those 
persons(s) to decide what methods will be used to identify the 
litany of possible risks. There have been numerous attempts in 
research to identify risk identification methodologies that span 
a variety of approaches such as formalized brainstorming 
workshops, the use of historical risk registers, and numerous 
expert elicitation methods. Maytorena et al. [8] provides an in- 
depth literature review of project risk identification methods, 
and points out the limitations with each of those methods 
commonly used. That research also explores how risk 
managers seek and gather information related to potential risks, 
and demonstrates that reliance on experience and historical risk 
registers may actually be counter-productive to proper risk 
identification due to an overreliance on a “check-list mentality” 
[8]. One reason for such behavior may be continued emphasis 
on past experience and various challenges that are generally 
within the control of project and risk managers. Because of this 
reliance on what has happened in the past, risk managers are 
prone to focus on the more common, higher likelihood, events. 



Unfortunately, this results in a lack of appreciation for events 
that, although considered very unlikely to happen, come with 
more severe consequences. 

F. Gray Swans and the Hidden Consequences 

In 2007 Nassim Nicholas Taleb popularized the concept of a 
“Black Swan” event by describing situations where an event 
occurs that is unprecedented in nature, thus impossible to some 
degree to anticipate, but that has catastrophic consequences [9], 
[10]. The metaphor refers to the fact that in historical European 
times phrases such as “as rare as a black swan” were used to 
express an impossibility, for at that point in time black swans 
were presumed not to exist. After such swans were 
subsequently discovered during Australian exploration, the 
phrase evolved to represent events that are thought to be 
impossible, but whose slim margin of likelihood do allow them 
to occur. When the associated consequences are severe, those 
events, despite being considered extremely unlikely to happen, 
can have far more impact than routine negative events that 
commonly occur. Although a generalization, the concept has 
roots in basic statistical inference when a proposed event 
cannot be confirmed to have zero likelihood probability, even 
with an infinite amount of observations of non-occurrence [10]. 
All it takes is a single occurrence to negate the assumption of 
impossibility. Unfortunately, that single occurrence could take 
place right when the associated negative consequences would 
cause the most disruption. 

From a project risk management perspective, black swans can 
and do take place. However, it is an impossibility itself to 
protect a project against their impact. Fortunately, the 
occurrence of such an event is inherently unlikely, so in the 
event something completely unprecedented and unforeseeable 
occurs the project would just have to proceed as best as 
possible. From this concept, a middle ground can be perceived 
- that there are events that are extremely rare and have 
happened, have significant consequence, but have 
undeterminable likelihood due to lack of knowledge or 
experience. Taleb refers to these as “Gray Swans”. The 
unfortunate aspect of these types or risks is that they can 
potentially be protected against, but are commonly dismissed 
due to their low likelihood of occurrence. These are the events, 
based on anecdotal evidence hoped to be confirmed by this 
research, that end up causing the most disruption to cost and 
schedule plans. Although their likelihood is low, resulting in 
managers dismissing them from concern, they do occur often 
enough to create problems. 

The intent of this paper is not to validate the concept of black 
or gray swans, but to use the latter concept to provide context 
on the potential biases present in risk identification activities. 

G. Perceived Biases 

Similar to many other subjective activities, risk identification is 
prone to various types of biases related to the understanding of 
occurrence and consequence. Various literature supports this 
claim, often by verifying that a large preponderance of risks 
identified are routine in nature and not representative of the full 


breadth of events that could take place [10] - [12]. Letens et al. 
describe this imbalance as a focus on the “Individual 
Dimension”. They documented these as 70% of the risks 
studied [11]. This term is defined to include the risks 
originating from within as opposed to outside. From a project 
perspective this is inteipreted as internal to the project versus 
external. Kolltveit et al. describe “External” sources of 
uncertainty separate from “Internal” sources of uncertainty by 
defining external to include the political situation, contractual 
variability, and other sources of variability and risk that can 
impact a project [12]. In contrast to internal sources such as 
design, internal organization, and internal goals, the external 
sources are largely outside of the control of the management 
personnel. The concept of internal vs. external is not just 
limited to a project itself, for risks may originate from both 
within the governing Agency itself or from elsewhere. Coonce 
et al., in an effort to better understand cost and schedule 
growth, used a root cause taxonomy that delineated events 
internal and external to NASA as well as internal and external 
to an individual project [13]. Events that could be external to a 
project but internal to NASA include events such as funding 
variability from Headquarters and guideline mandates from 
field Centers. Similar to the Letens et al. research, it is 
speculated here that a large preponderance of identified risks 
within past NASA projects are of the internal to the project 
nature, followed by external to project but internal to Agency, 
and then finally by those external to the Agency. However, it 
may very well be that the risks external to the project are the 
ones that actually cause disruption to cost and schedule. This 
dichotomy, along with an investigation of related behaviors 
stemming from temporal trends in risk identification, is the 
focus of this research’s methodology. 

VII. HYPOTHESES AND METHODOLOGY 
E. Hypotheses 

Based upon the literature in risk identification within project 
management, the author’s experience researching cost and 
schedule growth, and anecdotal observations by various project 
managers, a set of hypotheses can be established that will be 
used to guide an understanding of risk identification biases. 

The first hypothesis captures the sense that more focus is spent 
on identifying and worrying about risks that are the most 
familiar. These are typically the risks internal to a project, for 
they are based on past experience and can be somewhat 
mitigated by actions taken be the project team itself. Data 
analysis across a number of projects, combined with a risk 
categorization taxonomy, will provide insight into the support 
of this hypothesis. Demonstrating that this theory is significant 
would indicate that risks are identified with a bias towards 
internal concerns. Simply stated, this hypothesis can be phrased 
as: 

HI: Risks identified by a project are more often internal to a 
project than external. 



The second hypothesis aims to support that the risks that 
actually end up causing the most disruption are more 
commonly external than internal. This could be because 
internal risks are more easily mitigated upon identification, but 
also could indicate that events outside of the control of the 
project can and do pose a bigger threat. 

H2: The risks that become drivers of cost and schedule change 
are more often external to the project than internal. 

The combination of HI and H2, if both are supported, would 
underscore the belief that there is a disconnect between what 
types of risks are identified versus what risks are the bigger 
concern. A project may not be able to actively manage a risk 
external to their project, but an increased awareness would 
allow that project to better understand who, externally, could 
potentially mitigate the risk to the betterment of the project 
situation. To enable such, it is important to gain an 
understanding into how early a typical project actually 
identifies these types of events. 

The third and fourth hypotheses aim to gain an understanding 
of the temporal trends associated with identifying external 
risks. The thought is that these types of events can be identified 
as early as conceptual design, but are more commonly only 
identified once they become an active problem. Because of 
this, they are identified later and tracked for a shorter duration 
than other risks. The duration measurement is of interest 
because it is thought that external risks are often caught much 
closer to risk realization than internal risks. In other words: 

H3: Risks external to a project are not identified as early as 
risks internal to a project. 

H4: Risks external to a project are not tracked as long as risks 
internal to a project. 

Since hypotheses three and four are specific to only external 
risks regardless of their actual impact, investigating the 
temporal behavior of all risks that pose a cost and schedule 
problem, whether they are internal or external, may be 
worthwhile: 

H5: Risks that become drivers of cost and schedule change, 

regardless of whether they are internal or external, are not 
identified as early as other risks. 

F. Data 

The intent of this research is to substantiate anecdotal 
observations made when reviewing past NASA projects. To 
that end, actual NASA project data is leveraged in the form of 
monthly reports provided by field centers to NASA 
Headquarters in Washington, D.C. Although there is variability 
from one center to the next, and from one project to the next, 
there is some consistency in how each project’s risk status is 
documented each month. Among other data, each typically 
includes a standard 5x5 risk matrix with the top 5-10 risks 
identified and discussed. This information, in combination with 


temporal trends in cost and schedule plans, will provide insight 
into the hypotheses described above. 

NASA is divided into several high-level mission directorates; 
Human Exploration & Operations, Aeronautics, and Science. 
Within the Science Mission Directorate (SMD), there are four 
divisions based on the purpose of missions contained within; 
Earth, Heliophysics, Planetary, and Astrophysics. As with all 
Mission Directorates, SMD is spread amongst a number of 
centers across the country. Data for this research was pulled 
from the Jet Propulsion Laboratory (JPL) in Pasadena, 
California, The Goddard Space Flight Center (GSFC) in 
Greenbelt, Maryland, and the Langley Research Center (LaRC) 
in Hampton, Virginia. Within SMD, a centralized database is 
used at NASA Headquarters to document and store the monthly 
status reports provided by each of the field centers pertaining to 
development project status. This research is based on the data 
contained within that database, with permission having been 
obtained from the SMD Chief Engineer (representing 
ownership of the database) as well as key personnel from JPL, 
GSFC, and LaRC (representing ownership of the project- 
specific data). 

Data collection is still underway, with significant amount of 
time required to review each monthly status document to 
extract risk, cost, and schedule information (also cost, schedule, 
and mass margin detail in case helpful during future analysis). 
Based on a review of the available data within the database, 
thirty different missions have been identified that may have 
enough documented data to support analysis. Those 30 projects 
span the centers as well as all four mission divisions within 
SMD. To date, cost, schedule, and risk data has been gathered 
from two missions, and constitutes the data set for the 
preliminary findings contained within this paper. 

Because this research aims to identify and verify trends at the 
high level for wide-spread applicability to other industries and 
agencies, published results are kept at an aggregate level 
without specificity to any one project or center. 

G. Research Framework 

The methodology used will be a quantitative statistical 
evaluation of the various hypotheses discussed. To frame the 
analysis, a research framework has been developed and can be 
seen in Fig. 2. Within the research framework diagram there 
are four sets of variables defined. The “Risks Identified” 
represents the data gleaned from various projects across NASA 
SMD divisions and centers. These data include the top-level 
risks identified each month, along with monthly status 
measurements of project cost and schedule progress. Based on 
these data, when evaluated over time, temporal trends can be 
discerned which are accounted for in the “Temporal Trends” 
variable seen below. 




FIGURE 2. RESEARCH FRAMEWORK 


The first portion of analysis includes the assignment of a risk 
category specification to each risk. This assignment is based on 
the author’s interpretation of documentation associated with 
each project risk, and is based on a taxonomy described in a 
subsequent section. These categorization results, seen in the 
framework as “Risk Categorization”, represent dependent 
variables due to their dependability on the “influence of the 
independent variables” [14], although in this particular 
circumstance are also somewhat based on the author’s 
subjective interpretation. The final set of variables, “Project 
Cost & Schedule Change”, represents cost and schedule 
measurements pulled from the data sets. Although independent 
in the sense they are raw project data, they are considered 
dependent here relative to the risks and risk trends identified. 

H. Cost and Schedule Change Variables 

To understand which risks have correlation to cost and 
schedule change, the various cost and schedule measurements 
from the data sources are used. In each case a baseline cost and 
schedule can be gleaned from early project documentation, and 
can be used as a reference in which to gauge subsequent 
growth in either measurement. The primary cost measurement 
is the project’s total life cycle cost, which includes costs from 
Phase A through Phase E as defined in the NASA Systems 
Engineering Handbook [6]. A growth percentage is used to 
account for percent change from the baseline to subsequent 
levels. The schedule measurement is broken down into three 
milestone components; Preliminary Design Review (PDR), 
Critical Design Review (CDR), and Launch Readiness Data 
(LRD)/Launch. The latter milestone is considered readiness 
date during early design phases but becomes an actual launch 
milestone upon successfully mission launch. To quantify 
schedule growth a “months slipped” measurement will be 
tracked for each of the three schedule measurements. 

It is thought that the cost growth measurement will be more 
commonly indicative of cost and schedule growth since a 
number of missions, particularly planetary missions, have fixed 
launch windows due to mission opportunity windows related to 
orbital mechanics. However, both the cost and schedule growth 
measurements will be used as dependent variables within the 
research framework. 

I. Risk Characterization Taxonomy 

Within the literature there are numerous taxonomies used to 
understand the differences in risk and types of uncertainty [12], 
[13], [15]-[17]. Despite whether research is based on space 


systems, software development, or other industries, there seems 
to be a wide-spread delineation within the taxonomies between 
internal and external sources. Because of this, having any 
taxonomy used here include emphasis on such is important. 
Choosing a taxonomy that is at an appropriate level to support 
the data used within this research is also important, for the data 
is of a level typically reported to NASA Headquarters, as 
opposed to detailed case-study level research that could be 
garnered from an extensive review of a single project or two. 
Based on these needs, a taxonomy was chosen based on similar 
research performed by NASA Headquarters and the Aerospace 
Corporation [13]. That work focused on the characterization of 
root causes amongst past NASA projects, but with some small 
modifications the same taxonomy can be used here. Table 1 
below captures the modified taxonomy used to label each of 
the risks identified to date. 


TABLE 1. RISK CHARACTERIZATION TAXONOMY 


ID 

Agency 

Project 

Area 

Sub- Area 

PP1 

NASA Internal 

Project Internal 

Project Planning 

Design - Spacecraft 

PP2 

NASA Internal 

Project Internal 

Project Planning 

Design - Instrument 

PP3 

NASA Internal 

Project Internal 

Project Planning 

Technology - Readiness 

PP4 

NASA Internal 

Project Internal 

Project Planning 

Technology - Contingency/Backup 

PP5 

NASA Internal 

Project Internal 

Project Planning 

Cost and Schedule Plans 

PP6 

NASA Internal 

Project Internal 

Project Planning 

Programmatic 

PP7 

NASA Internal 

Project Internal 

Project Planning 

Heritage & Commonality 

PP8 

NASA Internal 

Project Internal 

Project Planning 

Other 

PEI 

NASA Internal 

Project Internal 

Project Execution 

Management - PM/SE/MA Growth 

PE2 

NASA Internal 

Project Internal 

Project Execution 

Management - Requirements Growth 

PE3 

NASA Internal 

Project Internal 

Project Execution 

Sys Development - Spacecraft 

PE4 

NASA Internal 

Project Internal 

Project Execution 

Sys Development - Instrument 

PE5 

NASA Internal 

Project Internal 

Project Execution 

Sys Development - Ingration & Test 

PE6 

NASA Internal 

Project Internal 

Project Execution 

Sts Developement - Ground Systems 

PE7 

NASA Internal 

Project Internal 

Project Execution 

Contractor Management 

PE8 

NASA Internal 

Project Internal 

Project Execution 

In-House Performance 

PE9 

NASA Internal 

Project Internal 

Project Execution 

Other 

AL1 

NASA Internal 

Project External 

Agency Level 

Annual Funding 

AL2 

NASA Internal 

Project External 

Agency Level 

Program Requirements 

AL3 

NASA Internal 

Project External 

Agency Level 

Component Supplier 

AL4 

NASA Internal 

Project External 

Agency Level 

Forced Budget Cap 

AL5 

NASA Internal 

Project External 

Agency Level 

Accounting Changes 

AL6 

NASA Internal 

Project External 

Agency Level 

Other 

CL1 

NASA Internal 

Project External 

Center Level 

Principles or Guidelines 

CL2 

NASA Internal 

Project External 

Center Level 

Workforce 

CL3 

NASA Internal 

Project External 

Center Level 

Estimating Process 

CL4 

NASA Internal 

Project External 

Center Level 

Center Facilities 

CL5 

NASA Internal 

Project External 

Center Level 

Other 

NE1 

NASA External 

N/A 

N/A 

Launch Vehicle 

NE2 

NASA External 

N/A 

N/A 

Congress/OMB 

NE3 

NASA External 

N/A 

N/A 

Force of Nature 

NE4 

NASA External 

N/A 

N/A 

Industrical Base Issue 

NE5 

NASA External 

N/A 

N/A 

Economic Issues 

NE6 

NASA External 

N/A 

N/A 

Partner Performance 

NE7 

NASA External 

N/A 

N/A 

Other 


As can be seen, there are several layers of delineation amongst 
the categories. First, there is a separation between risks internal 
to NASA and external to NASA. Risks internal to NASA 
include items related to project performance, development 
challenges, Agency level mandates, and so forth. Risks external 
to NASA include items such as launch vehicle delays (since 
NASA often contracts launch service providers that are prone 
to their own set of schedule conflicts), forces of nature, and 
nationwide economic variability. 

Within the NASA Internal set of risks, there is a separation 
between risks internal and external to a project. Internal project 
risks typically relate to project planning and performance, 
while external project risks include agency and center level 








requirements and implications. Within Project Internal the risks 
are separated into Project Planning and Project Execution. 
Within Project External the risks are separated into agency and 
center level risks. 

Because the original taxonomy used by the Aerospace 
Corporation [13] was specific to root cause analysis, several 
changes were made to adopt it to this research. First, a Project 
Planning item related to risk identification was changed to 
“Cost and Schedule Plans”. Secondly, within that same section 
a “Fleritage and Commonality” risk category was added to 
capture events where common hardware from other missions 
poses an issue. Within the Project Execution area a “Risk 
Mitigation” area was changed to “In-House Performance” to 
capture internal project risks related to underperforming in- 
house efforts. Within the Center Specific set of categories a 
“System Development” item was changed to “Center 
Facilities” to capture risks where center test facilities caused 
delays. Finally, a NASA External risk was added to capture 
partner performance. 

VIII. Data Analysis and results 

Since the intent of this paper is to describe the longer term 
methodology while including only some preliminary results, 
only analysis relating to hypotheses 1, 3, and 4 are discussed 
herein. After data collection from an increasing number of 
projects is completed, along with thorough review of each to 
understand causes of cost and schedule change, analyses 
relating to hypotheses 2 and 5 will begin. 

Between the two projects studied a total of 135 risks were 
identified and characterized per the taxonomy mentioned 
above. The split between the two projects was roughly even, 
with one project providing 56% of the risks while the other 
44%. Along with the assigned categorization of each risk, 
additional data in the forms of the starting month of each risk, 
the duration in months when each risk was tracked, as well as 
the average risk rank, likelihood value, and consequence value 
across those months was gathered. 

E. Hypothesis 1 

A first level understanding of HI (risks identified by a project 
are more often internal to a project than external) can be gained 
by a comparison of the number of external versus internal risks 
across the 135 data points. The results can be seen in Fig. 3, 
and are indicative that an overwhelmingly majority (116, 86%) 
of the risks are considered internal to the project. Outside of the 
project there are more Internal to NASA (15, 11%) than 
External to NASA (4, 3%). From this observation alone, based 
on this initial data set, it is clear that HI is supported. 



F. Hypothesis 3 

To investigate H3 (risks external to a project are not identified 
as early as risks internal to a project), a comparison is made of 
the average month, within a project’s life cycle, in which each 
of the types of risks were initially identified. Since different 
projects have different length development durations, and took 
place over a different set of years, the data was normalized 
such that a point in time was represented as a percentage 
between when risk tracking began and launch. This 
normalization allows a valid comparison of the points in time 
in which risks were initially identified during the development 
activity. 

Although statistical tests were planned to validate statistical 
significance of H3, from observations of the data the 
hypothesis not being supported became clear. For risks internal 
to projects, the mean value, as a percentage, of when they were 
identified during the project was 42%. For risks external to 
projects this same value was 39%, thus on average taking place 
earlier than the internal risks. 

G. Hypothesis 4 

To understand if H4 (risks external to a project are not tracked 
as long as risks internal to a project) can be supported, the 
average duration, in months, of each type of risk was evaluated. 
From a quick comparison, one can see that the mean duration 
of tracking for external projects (2.79 months) was shorter than 
internal projects (2.84). To check for statistical significance a Z 
test comparing two sample means with variance known was 
applied. 

In this case, the null hypothesis was assumed to be that 
internal project risk duration was less than or equal to external 
duration. By statistically rejecting the null, H4 could have 
been demonstrated. However, when assuming alpha values of 
either a=0.05 or a=0.1 the test value does not fall within the 
critical range, thus there is not enough evidence to reject the 
null. Although the mean value for external risks is smaller 
than that of internal risks, the difference is not enough to be 
statistically significant. 


H. Additional Analysis 

Using the same data collected and analyzed, statistical 
comparisons were also made to determine if the average 
likelihood or consequence values were greater for internal risks 
than external risks. This behavior would be indicative of a bias 
to consider the internal threats both more likely to happen as 
well as have the potential for greater impact to the project. 

In both cases the calculated mean values appeared to support 
the claim, with a mean value of 2.84 for likelihood and 3.54 for 
consequence for internal risks, compared to a mean value of 
2.79 for likelihood and 3.47 for consequence for external risks. 
However, when the same Z test was applied in each case there 
was not enough evidence to support the observations. 

I. Observations 

Based on these preliminary findings, the expected biases may 
exist within the data, but that additional data collection and 
analysis is warranted. Although HI was clearly demonstrated 
based on observation, neither H3 nor H4 were supported. HI 
confirmed that a large preponderance of risks identified were 
internal in nature, with only a small percentage being either 
external to the project or external to NASA altogether. This is 
consistent with literature suggestions that risk identification 
efforts do not always fully incorporate both internal and 
external concerns. 

Hypothesis 3 was not supported based on the observation that 
the mean identification month for external projects was not 
later than that for internal projects. This is inconsistent with 
expected results, but could be due to the smaller set of data 
currently being evaluated. 

Hypothesis 4 appeared to be supported based on initial 
observation, but was not shown to be significant. The same is 
said for additional analyses that evaluated the average 
likelihood and consequence ratings for each risk. Additional 
planned analysis using extended data sets may ultimately 
provide statistical significance in these areas. 

Hypotheses 2 and 5 will be evaluated in future analyses using 
additional review of current data as well as the incoiporation of 
additional data sets from a greater number of NASA projects. 

VI. Conclusion and Broader Applicability 

Risk management methodologies are commonly used 
throughout government space system design as part of systems 
engineering efforts to identify, track, and manage the various 
challenges and threats that each project faces. An underlying 
tenet to this research activity is that the realization of certain 
types of risks, and the biases that prevent them from being 
fully anticipated, causes just as much disruption to cost and 
schedule plans than other commonly stated root causes. By 
demonstrating that these biases, often relating to risks external 
to a project manager’s control, do exist and are impactful, 
future project and risk managers will be more prepared to 
acknowledge these biases. 


The research framework and methodology described herein is 
being formulated to guide a year-long investigation using 
extensive NASA historical project data. Thirty projects have 
been identified across three NASA Centers and all four 
Science Mission Directorate Divisions that will be potentially 
mined for supporting risk, cost, and schedule data. The result 
of this data collection and subsequent analysis will be 
thorough investigation into the biases speculated to exist. 

To date two of the thirty projects have been evaluated, 
providing 135 top-level risks that were of importance enough 
to be contained in monthly 5x5 risk status matrices. The 
results contained herein, with analysis findings documented 
related to hypotheses 1, 3 and 4, are based on these 135 data 
points. As additional data is collected and analyzed it is 
expected that these preliminary findings will evolve. In 
addition, subsequent causality analysis of risk realization to 
cost and schedule change will be incorporated, allowing 
thorough statistical evaluation of Hypotheses 2 and 5. 

Although data evaluated is NASA SMD data, biases verified 
within these activities are believed to be indicative of biases 
that exist throughout all complex large-system development 
efforts. 
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